Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database

Curated databases of scientific literature play an important role in helping researchers find relevant literature, but populating such databases is a labour intensive and time-consuming process. One such database is the freely accessible Comet Core Outcome Set database, which was originally populated using manual screening in an annually updated systematic review. In order to reduce the workload and facilitate more timely updates we are evaluating machine learning methods to reduce the number of references needed to screen. In this study we have evaluated a machine learning approach based on logistic regression to automatically rank the candidate articles. Data from the original systematic review and its four first review updates were used to train the model and evaluate performance. We estimated that using automatic screening would yield a workload reduction of at least 75% while keeping the number of missed references around 2%. We judged this to be an acceptable trade-off for this systematic review, and the method is now being used for the next round of the Comet database update

Findings of the WMT 2017 Biomedical Translation Shared Task

Automatic translation of documents is an important task in many domains, including the biological and clinical domains. The second edition of the Biomedical Translation task in the Conference of Machine Translation focused on the automatic translation of biomedical-related documents between English and various European languages. This year, we addressed ten languages: Czech, German, English, French, Hungarian, Polish, Portuguese, Spanish, Romanian and Swedish. Test sets included both scientific publications (from the Scielo and EDP Sciences databases) and health-related news (from the Cochrane and UK National Health Service web sites). Seven teams participated in the task, submitting a total of 82 runs. Herein we describe the test sets, participating systems and results of both the automatic and manual evaluation of the translations

Overview of the CLEF eHealth Evaluation Lab 2018

In this paper, we provide an overview of the sixth annual edition of the CLEF eHealth evaluation lab. CLEF eHealth 2018 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offered three tasks: Task 1 on multilingual information extraction to extend from last year’s task on French and English corpora to French, Hungarian, and Italian; Task 2 on technologically assisted reviews in empirical medicine building on last year’s pilot task in English; and Task 3 on Consumer Health Search (CHS) in mono- and multilingual settings that builds on the 2013–17 Information Retrieval tasks. In total 28 teams took part in these tasks (14 in Task 1, 7 in Task 2 and 7 in Task 3). Herein, we describe the resources created for these tasks, outline our evaluation methodology adopted and provide a brief summary of participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the lab tasks available for future research and development

Findings of the 2016 Conference on Machine Translation (WMT16)

This paper presents the results of the WMT16 shared tasks, which included five machine translation (MT) tasks (standard news, IT-domain, biomedical, multimodal, pronoun), three evaluation tasks (metrics, tuning, run-time estimation of MT quality), and an automatic post-editing task and bilingual document alignment task. This year, 102 MT systems from 24 institutions (plus 36 anonymized online systems) were submitted to the 12 translation directions in the news translation task. The IT-domain task received 31 submissions from 12 institutions in 7 directions and the Biomedical task received 15 submissions systems from 5 institutions. Evaluation was both automatic and manual (relative ranking and 100-point scale assessments)

Findings of the 2016 Conference on Machine Translation.

This paper presents the results of the WMT16 shared tasks, which included five machine translation (MT) tasks (standard news, IT-domain, biomedical, multimodal, pronoun), three evaluation tasks (metrics, tuning, run-time estimation of MT quality), and an automatic post-editing task and bilingual document alignment task. This year, 102 MT systems from 24 institutions (plus 36 anonymized online systems) were submitted to the 12 translation directions in the news translation task. The IT-domain task received 31 submissions from 12 institutions in 7 directions and the Biomedical task received 15 submissions systems from 5 institutions. Evaluation was both automatic and manual (relative ranking and 100-point scale assessments). The quality estimation task had three subtasks, with a total of 14 teams, submitting 39 entries. The automatic post-editing task had a total of 6 teams, submitting 11 entries

Automatisation des tâches documentaires dans un catalogue de santé en ligne.

Information Retrieval aims at enabling users to access the content of documents quickly and efficiently. In the medical domain, an increasing number of resources are available in electronic format, and there is a growing need for automatic solutions at several levels. Documents, and in particular texts, need to be selected and quality assessed to appear in catalogues; they need to be described with keywords and categorized withinmedical specialties to allow searches within the catalogues. These tasks are a challenge for Natural Language Processing, as they imply an "understanding" of the documents content by an automatic system. My PhD work has addressed this issue, and applied it to the automatisation of documentary tasks in a French online health catalogue, CISMeF. More specifically, this work has involved contributing to the enrichment of linguistic resources available in French for the medical domain, and developping systems for document watch and resource description. In this particular area, my focus was on MeSH automatic indexing with Main Heading/Sub Heading pairs.La Recherche d'Information a pour objectif de permettre aux utilisateurs d'accéder rapidement et efficacement au contenu d'une collection de document.Dans le domaine de la santé, le nombre de ressources électroniques disponibles augmente de manière exponentielle, et la nécessité de disposer de solutionsautomatiques se fait sentir à plusieurs étapes de la chaîne d'information. Les documents, en particulier les textes, doivent être sélectionnés selon des critères de qualité pour être inclus dans des catalogues; ils doivent également être décrits à l'aide de mots clés et catégorisés en spécialités médicales afin de faciliter les recherches effectuées dans les catalogues. Ces tachesconstituent un défi pour le Traitement Automatique de la Langue Naturelle car elles impliquent une "compréhension" du contenu des documents par un système automatique. Ce travail de thèse engage une réflexion sur la répartition des tâches documentaires entre l'homme et la machine dans le cadre particulier du Catalogue et Index des Sites Médicaux Francophones (CISMeF). A ce titre, il aborde l'automatisation des tâches documentaires dans le catalogue de santé en ligne CISMeF. Cette thèse apporte une contribution au développement de ressources linguistiques en français pour le domaine de la santé, et présente des systèmes de veille documentaire et de description automatiques de ressources de santé. Sur ce dernier point, l'accent a été mis sur l'indexation à l'aide de paires de descripteurs issues du thésaurus MeSH

Combining MEDLINE and publisher data to create parallel corpora for the automatic translation of biomedical text

International audienc

French MeSH Browser: a cross-language tool to access MEDLINE/PubMed

Poster at AMIA Annual Symposiu

A method of cross-lingual consumer health information retrieval

International audienc